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Abstract 



^ I The present collider data put severe constraints on any type of new strongly-interacting 

D . particle coupling to the Higgs boson. We analyze the phenomenological limits on exotic 

i-C ■ quarks belonging to non-triplet SU{3)c representations and their implications on Higgs 

searches. The discovery of the Standard Model Higgs, in the experimentally allowed mass 
range, would exclude the presence of exotic quarks coupling to it. Thus, such QCD particles 
could only exist provided that their masses do not originate in the SM Higgs mechanism. 



1 Exotic coloured fermions 



Exotic matter in higher representations of the SU{3)c colour group is an appeahng possibility 
which was already considered in the early times of QCD [THS]. In particular, the sextet repre- 
sentation has been extensively analyzed as a possible source of dynamical electroweak symmetry 
breaking [6^113]. It is well known that such exotic quarks modify very sizeably the running of 
the strong coupling and, therefore, their hypothetical existence is strongly constrained by the 
very successful experimental tests of asymptotic freedom |14j. 

Since not a single exotic QCD particle has been observed so far, their masses should be 
heavy enough to avoid the present experimental constraints from direct searches. However, even 
with very large masses, if those exotic quarks get their masses through the Standard Model 
Higgs mechanism, they would strongly enhance the production of Higgs bosons at LHC. The 
non-decoupling character of the Higgs couplings, being proportional to the coupled-object mass, 
implies sizeable effects from any strongly-interacting heavy mass scale generated by the Higgs 
mechanism. Therefore, the present collider limits on the production cross section cr{gg — )• H) 
put a very severe constraint on the possible existence of such objects. 

Let us consider an exotic spin-^ fermion Xr, with mass Mx, belonging to the irreducible 
representation R = (Ai, A2) of SU{3)c- The dimension of the representation is given by (Ir = 
I (Ai + 1)(A2 + l)(Ai + A2 + 2); the fundamental 3 = (1,0) [3* = (0, 1)] and adjoint 8 = (1, 1) 
representations have dimensions dp = 3 and dA_ = 8, respectively. The gluonic couplings of 
Xr are fixed by the generators (a = 1, • • • , c^a), satisfying [i^, t^] = i/"*'^ t^. The quadratic 
Casimir operator. 



E 

a=l 



t% t\ = Cr 1,^ , Cr = ]-(\l + Xl + Ai A2 + 3Ai + 3A2) , (1) 



determines the trace normalization factor for the representation R;. 

-^[tuA) = Tr5'^\ Tr = (2) 

This trace factor grows rapidly with increasing dimensions d/j, implying larger contributions of 
the exotic object Xr to the relevant QCD cross sections: Tp = i, Tg = |, = 3, Tio = 
Tis = 10 . . . , where 6 = (2, 0), 10 = (3, 0), 15 = (2, 1) ... 

If kinematically allowed, charged exotic quarks would be copiously produced in e^e~ anni- 
hilation. For a charged Xr the ratio Re+e- = cr{e^e~ — )• hadrons)/(T(e'''e~ — )• fJ-^f^') would raise 
dramatically at the production threshold s = 4Mj^ with an additive contribution Ai?g+g- = 
dnQx^Qco- ^ neutral exotic would be pair-produced at 0{a'^) through gluon emission, i.e. 
e+e" — )• qqg — )• qqXj^Xj^. Independently of their electric charge, exotic quarks would imply 
large modifications of the hadronic cross sections at pp and pp colliders and a proliferation of 
new hadrons containing Xr constituents (unless the Xr lifetime is too small to hadronize) . The 
absence of any exotic signal in the present data puts the lower limit on the mass Mx well above 
100 or 200 GeV. 

New fermions in higher QCD representations would contribute to the QCD f3 function 

= as(3{as), (3{as) = J2 ■ (3) 

At the two loop level [T5l[T6] . 



R 

h = -^CI + \Y.^M^Ca + 3Cr), (4) 
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Figure 1: Ratio cr{pp — >■ XjiXji)/(T{pp -> gg) at ^/s = 7 TeV, as a function of Mx- The different curves 
correspond to the exotic fermion Xn in the sextet (continuous, blue), octet (dashed, violet), decuplet 
(dotted, black) and 15 (dash-dotted, red) representations. 



where is the number of fermion flavours in the representation R. In the three-generation 
Standard Model {np = 6) both /3i and /32 are negative. In order to flip the sign of /3i (/32), 
np > 16 (8) triplet quarks would be needed. However, the larger algebraic contribution of a 
higher colour representation implies a much faster lost of asymptotic freedom. Keeping np = 6, 
the only possible additions preserving /3i < are at most two sextet or one octet fermion 
representations; but even a single sextet flips already the sign of /32- Since the running of ag has 
been successfully tested with high precision (at the four loop level) from the r mass scale |17lll8j 
up to energies above 200 GeV [14J, exotic quarks in higher QCD representations are clearly 
excluded in this energy domain [T2H23]. 

Higher energy scales are presently being explored at the LHC, where the main production 
mechanism of exotic QCD fermions is gg — t- XjiXji, with a subdominant contribution from 
qq — )■ XrXji. The calculation of the corresponding partonic cross sections is straightforward at 
tree level; we obtain 



aigg XrXr) 
a{qq XrXr) 



where 
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in agreement with Ref. [23]. Particularizing to the fundamental representation, one gets the 
well-known results for quark-antiquark production [25j. The production of exotic fermions in 
higher representations is enhanced by the global algebraic factor ^r = CrcIr/ {Cpdp) [^g = 5, 
^8 = 6, = 15, = 20, . . .], which is further reinforced by another factor Cr/Cp in the 
leading parts of the 2-gluon contribution. Figure [1] shows the ratio (T{pp — t- XrXr) / a{pp — )• qq) 
at ^/s = 7 TeV, as a function of Mx, for the representations with lower dimensions. We 
have convoluted the partonic cross sections with standard parton distribution functions and 
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have assumed a common K factor for all representations; i.e., we have taken the same QCD 
corrections as for triplet quark production. This is a very conservative assumption because, given 
the larger algebraic factors, gluonic corrections should be larger for higher colour representations. 
Thus, the curves in Fig. [1] are actually lower bounds on the expected production ratios. The 
enhancement factors are predicted to be larger than 10 for sextet and octet fields and much 
higher values are obtained for higher-dimensional representations. 

Once produced, the exotic Xji particles should decay strongly generating an excess of (multi) 
jet events. Fermionic objects in the triplet, sextet and 15 representations could couple to a qg 
{qg) operator and are thus expected to produce 2-jet events, while fermionic octets and decuplets 
have qqq {qqq) quantum numbers and should be looked for in 3-jet events The generic 2-jet 
searches performed at the LHC [26p27j have not found any evidence for new particle production, 
severely constraining narrow resonances decaying into gg, qg or gg final states. The lower limits 
on different types of strongly-interacting particles have been pushed up beyond the 1 TeV scale; 
for instance the data excludes at 95% CL excited quarks with mass below 2.64 TeV or coloured 
octet scalars with mass below 1.92 TeV. Searches with 3 jets have been already performed by 
CMS [28] and CDF |29j; no significant excess has been found, excluding gluino masses up to 280 
GeVir 

A dedicated search for stable quarks in higher colour representations was performed longtime 
ago by CDF No such particles were found in 26.2 nb~^ of data; at 95% CL, the resulting 
lower limits for Mx were 98 (84) GeV for color sextets, 99 (86) GeV for octets, and 137 (121) 
GeV for decuplets, assuming that Xn carries charge one (either one or zero). A recent CMS 
search for heavy stable charged particles produced at LHC has put a lower limit of 808 GeV 
(95% CL) on a stable gluino, under the conservative hypothesis that any hadron containing this 
particle becomes neutral before reaching the muon detectors (relaxing this hypothesis, the limit 
improves to 899 GeV) [35]. Slightly weaker bounds have been set by ATLAS through a search 
for slow- moving gluino-based R-hadrons [36] . 

The present 95% CL limits on fourth- generation quarks, my > 372 GeV [37] and rrit' > 
404 GeV I38j (mj/ > 340 GeV |39lll0]), assume the decays b' Wt and t' Wb {f 
Wq), respectively. While these direct limits are set on new triplet quarks, the (absence of) 
experimental signature, W + Jets, is also sensitive to other strongly-interacting exotic particles 
in weak SU{2)l representations, as we are going to consider next, provided they decay within 
the detector through Xr — > WX'j^ W + Jets. 



2 Higgs production at LHC 

In the Standard Model, the Higgs mechanism is responsible for all particle masses. If the mass 
of the exotic colour object Xr is also generated through its coupling to the Higgs boson, the 
Higgs properties are modified through quantum loops involving the fermion Xji. Let us consider 
the consequences of a generic Higgs coupling 

Ch = -ex ^ H{x) [Xr{x)Xr{x)] , (8) 

with V = {V2Gf)~^/'^ = 246 GeV the Higgs vacuum expectation value. The usual Standard 
Model mechanism for fermion masses gives cx = 1- It requires an electroweak SU{2)l doublet, 
i.e., two exotic Xr bosons differing by one unit of electric charge and with masses degenerated 
enough to satisfy the electroweak precision tests. One should also implement the cancelation 

^Stronger bounds on gluino masses are obtained through searches for jet events with large missing energy or 
transverse momentum :30r,33j . The excluded region depends on the assumed supersymmetric model, reaching in 
some cases the 1 TeV scale. 
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Figure 2: Ratio cr(g.g H)/asM at ^ = 7TcV and Mx = 500 GeV, as a function of Mx- The different 
curves correspond to an exotic fermion multiplet Xji in the sextet (continuous) , octet (dashed) , decuplet 
(dotted) and 15 (dash-dotted) representations. 



of the electroweak anomalies generated by the new SU{2)l doublet, but there are well-known 
ways to do it (for instance adding exotic leptons) without modifying our subsequent arguments. 

A different possibility would be an exotic coloured SU{2)l singlet, coupling to the Standard 
Model Higgs doublet ^{x) through the dimension-5 operator ($1^$) {XrXr). In the unitary 
gauge, the fermion bilinear would be coupled to a factor {v+HY'; i.e., Mx would be proportional 
to the Higgs vacuum expectation value squared (over some dynamical scale) and cx = 2. 

Since Xn couples strongly to gluons, the vertex ([8|) generates a very sizeable contribution 
to the main Higgs production channel at LHC, through an intermediate X^X^ virtual pair: 
gg — )• XnXn — )• H. The resulting amplitude can be easily obtained from the standard quark- 
loop result, accounting for the different colour factors. There would be two equal (for degenerate 
masses) contributions from a SU{2)l doublet of Xr fermions with cx = 1, while cx = 2 in the 
SU{2)l singlet case. Therefore, the two scenarios give the same result: 

2 

5{s-Ml), (9) 
where 

C — 4 arcsin^ (1/y^) , x>l 
^(x) = - [4 + (x-l)/(x)] , /(x)= r ,2 . (10) 

The first term in ([9]) is the usual triplet-quark contribution; it is completely dominated by the 
top loop because the function J-{x) vanishes in the massless limit {x — )• 0). The second term 
stands for the additional contribution from an (electroweak singlet/doublet) exotic coloured 
fermion Xr. Given the experimental constraints on Mx discussed before, Mfj < AM^ in the 
interesting kinematical regime and the corresponding loop function does not have any absorptive 
part. Moreover, the numerical result is not sensitive to the exact value of Mx because J-{x) is 
a very smooth function for x > 1, decreasing gently from = 2 to J-"(oo) = 4/3. 

Owing to relative colour enhancement factor Tn/Tp, the Xn contribution generates a large 
increase of the Higgs production cross section. The ratio a^gg — )■ H)/asM for different colour 
representations is shown in Fig. [21 as a function of Mh, taking ^/s = 7 TeV and Mx = 500 
GeV. The normalization usm = (^{§9 — ^ ^^)sM is the Standard Model cross section with three 
quark families. Again, we have assumed the same QCD corrections as for triplet quarks, which 
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underestimates the actual cross section. Very large enhancement factors are obtained for all 
non-triplet representations. In the sextet and octet cases, the Higgs production cross section is 
larger than the SM one by a factor between 40 or 300, depending on Mh- The enhancement 
surpasses the three orders of magnitude for the 15 and higher colour representations. 



3 Higgs search 



Since the decay H — t- XhXr is not kinematically allowed for Mh < 2Mx, a heavy Higgs would 
decay into WW, ZZ and with approximately the same branching fractions as in the absence 
of the fermion X^. The Standard Model Higgs has already been experimentally excluded for 
Higgs masses between 2Mw and 600 (525) GeV, at 95% CL (99% CL) The existence 

of an additional coloured fermion would only make the exclusion much stronger. More care 
has to be taken below the WW threshold, because the same enhancement present in the Higgs 
production cross section also appears in the H ^ gg decay width, modifying all branching ratios. 
Figure |3] shows the total Higgs decay width Th, as a function of Mh-, for the Standard Model 
with three families of triplet quarks, and with the addition of one (electroweak singlet/doublet) 
colour sextet or octet multiplet. The exotic contributions are small for Mh > 2M]v, but at lower 
Higgs masses they generate a big enhancement of Th ■ Figures IH [5] and [6] plot the corresponding 
branching ratios in the different channels. 
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Figure 3: Higgs total decay width in the 3- 
generation Standard Model (SM) , and with the 
addition of colour sextet (SM6) or octet (SMS) 
multiplets. 
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Figure 4: Higgs decay branching ratios in the 
3-generation Standard Model. 











WW 


7Z 






hb 








0.1 












0.01 




V - — 








0.001 
























10-' 













100 150 



MuiGeV) 



Figure 5: Higgs branching ratios with the ad- Figure 6: Higgs branching ratios with the ad- 
dition of a colour sextet multiplet. dition of a colour octet multiplet. 

The strong enhancement of the two-gluon decay channel at low Higgs masses, affects in a 



5 



30 
25 
20 

> 

10 
5 

100 110 120 130 140 150 

M//(GeV) 

Figure 7: Rww,zz at y/s — 7 TeV, as a function of Mh, with the addition of colour sextet (SM6) or 
octet (SMS) multiplets. 

very sizeable way the suppressed (one loop) 27 and jZ decay modes, making them insignificant. 
However, in the WW and ZZ modes the branching fraction suppression cannot compensate 
the large enhancement of the production rate. In order to compare with the LHC experimental 
data, the relevant ratio is 

^ ajpp ^ H) BijH ^ VV) 

where SM refers again to the Standard Model with three quark families. This is plotted in 
Fig. [71 for sextet and octet colour representations, showing that, at = 7 TeV, Rvv > 15 in 
the relevant range of Higgs masses. Much larger values of Rvv would be obtained with higher- 
dimensional representations. Therefore, the present ATLAS [5T| and CMS [12] searches in the 
WW and ZZ channels, already exclude a Standard Model Higgs boson coupled to exotic colour 
multiplets, in the whole range between 110 and 600 GeV. 

The combined CDF and DO data [l3] exclude Higgs masses between 100 and 108 GeV (95% 
CL), within the three-generation Standard Model. Although gg ^ H accounts for 76% of the 
Higgs production cross section in this mass region, the Tevatron constraints are mainly extracted 
from qq — t- WH/ZH, with a small contribution from qq — )• q'q'H. These production mechanisms 
are not enhanced by the exotic colour- multiplet contributions. In this mass range the main Higgs 
signature is -ff — )> bb; therefore, the Tevatron information translates into 95% CL upper bounds 
for TZf^i = Bi{H bb)/BT{H bb)su ranging from 0.45 at 100 GeV to 1.1 at 110 GeV [43j. 
The addition of a sextet (octet) multiplet implies TZf^i values ranging from 0.33 (0.26) at 100 
GeV to 0.31 (0.24) at 110 GeV, which are slightly below the present Tevatron bounds. A mild 
improvement of the Tevatron constraints could exclude sextet ot octet contributions for Mh 
between 100 and 110 GeV. 

The LEP exclusion limit below 114.5 GeV [H] needs also to be re-analyzed in view of the 
strong enhancement of Br(i7 — )• gg). While the production mechanism e'^'e" — )■ Z* — )■ ZH 
remains unchanged in the presence of exotic quarks, there is a large suppression of the Higgs 
branching fractions into bb and t~^t~ and, therefore, of the sought experimental signal. OPAL 
performed a generic search for neutral scalars decaying into an arbitrary combination of hadrons, 
leptons, photons and invisible particles, covering as well the possibility of a stable scalar |45| . 
Thus, the OPAL bound, Mh > 81 GeV (95% CL) [15], remains valid in the presence of exotic 
colour multiplets. For larger masses, the combined LEP analysis relies in the H ^ bb decay 
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Figure 8: The LEP exclusion limits on Br(i? — >■ bb) [44^, as a function of Mh, are compared with the 
expected signals in the presence of one exotic (electroweak singlet/doublet) sextet (top red curve) or octet 
(bottom blue curve) multiplets. 

mode. Figure [5] compares the LEP bounds on Br(ff — )• bb) |44j . with the expected values with 
one (electroweak singlet /doublet) sextet (top red curve) or octet (bottom blue curve) multiplet. 
Higgs masses below 96 (92) GeV are then excluded in the sextet (octet) case. Note, however, that 
additional exotic multiplets or higher colour representations would imply a larger suppression 
of Br(ff — )• bb), weakening the LEP and Tevatron constraints. 

The triplet case of a fourth quark generation has been already discussed before [46H58j . The 
enhancement of cr{gg — )• H) is milder, about a factor of 9, but enough to exclude Higgs masses 
above 110 GeV from the LHC constraints on Ryv- The corresponding weaker enhancement of 
Br(iJ — )• gg) implies a much smaller suppression of the remaining channels; in particular, for 
Higgs masses smaller than 110 GeV, the bb branching fraction is predicted to be above the LEP 
bound in Fig. [HI Therefore, in the presence of an additional (electroweak singlet /doublet) colour 
quark triplet, the Higgs boson is excluded in the whole mass range up to 600 GeV. 

4 Discussion 

Present LHC data imply that a Standard Model Higgs cannot exist in the presence of new 
coloured fermions coupled to it, in exotic QCD representations, except for a small Mh region 
between 92 and 110 GeV which could be soon excluded. Exotic quarks in higher-dimension 
colour representations generate a very large enhancement of cr{gg — )■ H), in contradiction with 
the available experimental bounds. Strong limits have been already put before in the case of 
a fourth quark generation, where the enhancement of the Higgs production cross section is 
milder gHHSO]. 

One could certainly try to evade the experimental constraints, enlarging the Standard Model 
in appropriate ways to compensate the enhancement from exotic quarks. For instance, intro- 
ducing additional coloured scalars with couplings to the Higgs adjusted to suppress the gg ^ H 
amplitude [59H63] . Another possibility is "hiding" the Higgs; i.e., opening new decay channels 
into invisible modes without strong interactions [52il531[MU69j . in order to suppress the visible 
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branching fractions. While well-motivated arguments, such as dark matter, exist to do it, we 
feel that this hides the main reason behind such strong exclusion: the intrinsic non-decoupling 
of the Yukawa vertex ([8]) makes the Higgs boson sensitive to arbitrary high mass scales. 

The Higgs vacuum expectation value is linked to the electroweak scale, i.e., to the gauge 
boson masses M\y and Mz- In the Standard Model this scale is also used to generate all 
fermion masses through the Yukawa couplings. The known pattern of lepton and quark masses, 
with very different mass scales, implies a large variety of Yukawa couplings with magnitudes 
ranging from m^/v ~ 10~^^ to nit/v ~ 0.7. This wide range of couplings/scales is not yet 
understood. Introducing additional fermions with even higher masses, would bring much larger 
Yukawa couplings inducing a non-perturbative dynamical regime in the electroweak sector. In 
fact, the Higgs production and decay amplitudes used in our analysis are subject to potentially 
large electroweak corrections [58j . 

If a light neutral scalar boson is finally discovered, one should study very carefully its proper- 
ties in order to clarify the true pattern of electroweak symmetry breaking. The Standard Model 
is certainly a very plausible possibility, but heavier mass scales should not couple to the Higgs 
boson, i.e., they should have a different origin. Multi-Higgs models offer a much more flexible 
framework to accommodate future data, but soon or later they would also face the characteristic 
non-decoupling of the Higgs mechanism in (parts of) their extended Yukawa couplings. A per- 
haps more interesting possibility is that fermion masses could be generated through a mechanism 
different than the one responsible for the gauge boson masses. Another alternative, of course, is 
that the Higgs boson does not exist (dynamical symmetry breaking) or it is a composite object 
with rather different properties. The forthcoming LHC data should soon show us the option 
chosen by Nature to break the electroweak symmetry and hopefully provide some hints on the 
dynamics behind the observed pattern of fermion masses and mixings. 
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